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This study explored the usefulness of a classroom assessment technique called the 
activity summary template. It is proposed that the activity summary template enables 
students to process and organize information learning during an investigation. This 
process will in turn help students to achieve greater learning outcomes. The activity 
summary template is comprised of a summary of the main ideas, a statement of the 
importance of the activity for answering a research question, and new questions that 
emerge from the activity. The study took place in the context of the summative 
evaluation of Astronomy Village®: Investigating the Solar System™. Using a path 
diagram analysis, the results indicate that the activity summary template holds great 
promise for helping low-achieving students to accomplish greater learning 
outcomes. In addition, it is a useful tool for teachers to monitor how well students 
are processing and organizing the information they are learning. 



OBJECTIVES AND SIGNIFICANCE 

McGee et. al. (2001 ) demonstrated the overall effectiveness of the NSF-funded Astronomy 
Village: Investigating the Solar System. Students significantly increased both their understanding 
of complex solar system concepts as well as problem-solving abilities related to analyzing data and 
drawing conclusions (Dimitrov, McGee, & Howard, 2002). The results also indicate that student- 
learning gains on complex content were significantly greater than their gains in problem solving. 
Prior research indicates that an important factor in successful problem-solving performance is a 
well-organized knowledge base (Shin, Jonassen, & McGee, in press). This would suggest that in 
order to improve problem-solving performance, teachers using Astronomy Village should help 
students develop a well-organized knowledge base. In this paper we explore a classroom 
assessment technique, called activity summaries, that could become a useful tool for monitoring 
how well students are organizing new information that they process during an extended 
investigation. 



THEORETICAL FRAMEWORK 

There are three generic process skills within the activity summary template: summarizing, 
synthesizing, and question generation. After completing an activity or set of related activities in an 
extended investigation, a student completes a generic template. In this template students summarize 
the main ideas in the activity using three sentences, state the importance of the activity in helping to 
answer the research question, and generate new questions. 

The activity summary template was developed as a means to track student progress during 
extended investigations (McGee & Howard, 1998). It has been used extensively in research related 
to the Center for Educational Technologies® Astronomy Village programs. With Astronomy Village 
students spend approximately four weeks engaging in extended investigations on cutting-edge 
astronomy topics. Early evaluation work on Astronomy Village demonstrated that students have a 
difficult time synthesizing the results of related activities. For the most part they can successfully 
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complete individual activities, but they often fail to understand the connection between the individual 
activity and the research question. This had important implications for the extent to which students 
can integrate new information into a well-organized knowledge base. 

The activity summary template helps teachers diagnose where students are having difficulty. 
The coding scheme for the main ideas measures the extent to which students identified the most 
relevant concepts. If students were not selecting the most relevant concepts as being important, it 
would be extremely difficult to synthesize those concepts with the overall investigation. The coding 
scheme for the statement of importance measures the extent to which the student connects a relevant 
main idea to the overall research question (see Appendix A for an example of one highly rated and 
one poorly rated activity summary). The ratings of the statement of importance have been used 
successfully to evaluate the efficacy of curriculum changes in the implementation of Astronomy 
Village ® (McGee & Howard, 1999b). The results point to the kinds of supports that students need 
to synthesize learning activities with the overall research question. 

In this study we were interested in two questions: (1) What factors influence the quality of 
the summaries and the statements of importance? (2) How do the summaries and statements of 
importance influence student learning outcomes? We investigated these questions in the context of 
the Astronomy Village summative evaluation. 

ASTRONOMY VILLAGE. INVESTIGATING THE SOLAR SYSTEM 

Through Astronomy Village students are transported to a virtual village in Hawaii where 
they investigate one of two core research topics: what the surface of Pluto might look like when the 
first NASA mission arrives in 2015, or the search for life in the solar system (McGee & Howard, 
1999a). The program is designed such that a virtual mentor guides students in completing multiple 
investigation cycles that mirror the phases of scientific inquiry. 

In the first investigation cycle students are introduced to the core research question 
concerning either the surface of Pluto or the core requirements for life. During the exploration 
phase of the investigation, students see the types of data they will use in the investigation to prepare 
them for future analyses. In the background research phase students read library articles and listen 
to lectures to help them understand key background concepts. During the data collection and 
analysis phases students use the results of their analysis to draw conclusions about the research 
question. Students complete the investigation by hosting a virtual press conference. A virtual press 
corps asks them questions about the investigation the students just completed. This core 
investigation cycle lasts about one week. 

Students then follow the same sequence of phases as they did in the core investigation when 
they undertake a focused investigation on a narrower topic. For example, students may investigate 
whether icy volcanoes could exist on Pluto by examining the surfaces of icy moons in the solar 
system. Or students may examine temperature-pressure relationships on a variety of planets and 
moons to determine where the conditions are right to support liquid water. 

Teachers using Astronomy Village have adopted one of two basic approaches. In the first 
students complete the core research project and then complete each of the focused investigations 
related to the core research project. In the case of Search for Life, there are four focused 
investigations. In the second approach students complete the core research project and then 
complete just one focused investigation. The teacher ensures that each of the focused investigations 
has at least one project team working on it. The students then host a press conference for their peers 
so that all of the students can learn the content in each module. 
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ASSESSMENT INSTRUMENT 



Three principles guided the design of the assessment instrument. First, the assessment 
instrument should reflect important thinking and problem-solving skills from the discipline of 
planetary science (Hickey, Wolfe, & Kindfield, 1999; Sheppard, 2000). In Astronomy Village ® 
students investigated authentic questions, such as whether liquid water exists in the solar system. 
These require important thinking and problem solving skills from the discipline of planetary 
science. Therefore, we achieved this principle by designing assessment tasks that reflect the 
thinking and problem solving that Astronomy Village targets. 

The second guiding principle was measuring the extent to which students transfer their 
thinking and problem-solving skills into new contexts (Bransford, Brown, & Cocking, 1999). This 
principle reflects the philosophy that a critical aspect of education is whether learning transfers 
(Sheppard, 2000). When there is no specific transfer situation, the assessment becomes the transfer 
situation (Hickey, Wolfe, & Kindfield, 1999). Astronomy Village supported transfer by having 
students investigate critical processes and features on a variety of planets and moons. For the 
assessment instrument students had to transfer their understanding to hypothetical planets and 
moons. 



The third guiding principle was ease of administration and scoring for the target population. 
In prior research at the high school level, we have had success measuring complex problem-solving 
and argumentation abilities using an extended response format (Shin, Jonassen, & McGee, in press; 
Hong, McGee & Howard, 2001). However, at the middle school level there was concern that the 
extended response format would better reflect students’ writing abilities than their problem-solving 
abilities. In addition, the extended response format was too labor intensive to score within the 
budget limitations of the project. We therefore chose to use a machine-readable multiple-choice 
format. Taking into account the three guiding principles collectively, we felt confident in developing 
an assessment instrument that would measure important learning outcomes in a cost-effective 
manner. 

We identified the key complex content ideas that were presented in each of the nine 
investigations within Astronomy Village along with the key problem-solving skills related to 
drawing conclusions from data and inferring planetary processes from analyzing images of surface 
features. We contracted with item writers to develop the assessment items related to the underlying 
concepts within the investigations. The resulting instrument has four subscales: Search for Life 
complex content, Search for Life problem solving, Mission to Pluto complex content, and Mission 
to Pluto problem solving. This study focused on the Search for Life complex content and problem- 
solving subscales. 



INVENTORY OF METACOGNITIVE SELF-REGULATION 

Students also completed the Inventory of Metacognitive Self-Regulation (IMSR), which 
measures five factors related to awareness of learning processes and control of learning strategies 
(Howard, McGee, Shia, & Hong, 2001). The IMSR includes 32 items that use a five-point Likert 
scale. For each of the 32 items, students are instructed to circle the answer that best described the 
way they solve problems in math or science class (l=never, 2=seldom/rarely, 3=sometimes, 
4=often/frequently, 5=always). The validation of the IMSR is discussed elsewhere (Howard & 
McGee, 2000). The five factors are: 

• Knowledge of cognition. How much do learners understand about their unique cognitive 
abilities and the ways they learn best. Includes an awareness of one's own learning and memory 
processes as well as learning strengths and weaknesses. 

• Objectivity. Do learners stand outside themselves and think about their learning as it proceeds. 
Includes an awareness of their learning goals and alternative choices in accomplishing a 
learning goal. 
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• Problem representation. Are learners aware of strategies they use to understand the problem 
fully before proceeding. 

• Subtask monitoring. Do learners break the problem down into subtasks and monitor the 
completion of each subtask. 

• Evaluation. To what extent are learners aware of checking their work throughout the entire 
problem-solving process to evaluate if it is being done correctly. 

METHOD 

While completing the modules in Astronomy Village ®, students completed summaries of 
many of the articles and activities in the module. Each student completed from 3-5 activity 
summaries. Two independent raters judged the quality of the summaries and statements of 
importance. Any disagreements in rating were reconciled through discussion. Interrater reliabilities 
using Cronbach’s alpha were greater than 0.70 for both the summaries and the statements of 
importance. For all of the subsequent analyses, each student was given one score for their 
summaries and one score for their statements of importance based on the average of each factor 
across all of the summaries they completed. 



RESULTS 

To answer the question of what factors affect the quality of the student activity summaries, 
we used regression analysis to test the following pretest factors: content understanding, problem 
solving, and metacognitive self-regulation (See Table I .) For the quality of the summaries there 
were only two metacognitive self- regulatory factors that were statistically significant- — knowledge of 
cognition (6 = ,34, P < .001) and objectivity (6 = -0.36, P < .001). These two factors accounted for 
20, 1 percent of the variance. Neither content understanding nor problem solving were statistically 
significant predictors. 

The results for the quality of the statements of importance were similar, except instead of 
knowledge of cognition, problem representation (6 = .27, P < .001) and objectivity (B = -0.35, 

P < .00 1 ) were significant predictors, accounting for I 1 .9 percent of the variance. Once again, 
neither content understanding nor problem solving were statistically significant predictors. 

It is interesting to note that achievement-related variables were not significant predictors of 
activity summary performance. The fact that different metacognitive self-regulation components 
predict the quality of the summaries and statements of importance indicates that these are distinct 
process skills. This conjecture is confirmed by looking at the correlation between the main ideas 
and statements of importance (r = 0. 1 2). 
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Table 1 : Standardized Beta Coefficients for Statistically Significant Predictors of Activity Summary 
Quality 





Main Ideas 


Significance 


R 2 


20. 1 % 


1 1 .9% 


Content 


• 


• 


Problem solving 


• 


• 


Knowledge of cognition 


0.34 


• 


Objectivity 


-0.36 


-0.35 


Problem representation 


• 


0.27 


Snbtask monitoring 


• 


• 


Evaluation 


• 


• 



In order to investigate how the quality of the activity summaries influences student learning 
outcomes, we used regression analysis, controlling for the influence of pretest performance. The 
results indicate that the activity summaries are significant predictors of both content as well as 
problem-solving learning outcomes (see Table 2). After controlling for pretest performance on 
content understanding, the quality of the summaries (6 = 0.3 1 , P < .00 1 ) and the statement of 
importance (B = 0. 1 3, P < .05) were significant predictors of content understanding posttest 
performance. In addition, knowledge of cognition (6 = 0. 1 8, P < .0 1 ) was a significant predictor. 
These four factors accounted for 63.9 percent of the variance. 

After controlling for pretest performance on problem solving, the quality of the summaries 
(B = . 13, P < .05) and the statement of importance (6 = .16, P < .01) were significant predictors of 
problem-solving posttest performance. In addition, problem representation (6 = 0. 14, P < .05) was a 
significant predictor. These four factors accounted for 76.2 percent of the variance. 
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Table 2: Standardized Beta Coefficients for Statistically Significant Predictors of Posttest 
Performance 





Content Understanding 
Posttest 


Problem-Solving Posttest 


R 2 


63.9% 


76.2% 


Pretest 


0.40 


0.62 


Main ideas 


0.31 


0.13 


Significance 


0.13 


0.16 


Knowledge of cognition 


0.18 


• 


Objectivity 


• 


• 


Problem representation 


• 


0.14 


Subtask monitoring 


• 


• 


Evaluation 


• 


• 



Figure I shows a path diagram summarizing the results of the combined regression 
analyses related to content understanding. 
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Figure 2 shows a path diagram summarizing the results of the combined regression 
analyses related to problem solving. 




Figure 2: Path analysis of Search for Lite problem solving. 



CONCLUSION 

The results of both path diagrams demonstrate the importance of the activity summary 
template for helping students develop both content understanding and problem solving. For content 
understanding, the influence of the activity summary template was equivalent to the influence of the 
pretest performance. For problem-solving performance the influence of the activity summary 
template was significantly less than the influence of the pretest performance. The results for 
problem solving are consistent with prior research on Astronomy Village ® (Dimitrov, McGee, & 
Howard, 2002). Students achieved a significantly greater learning outcome on the content 
understanding subscale than on problem solving. In other words, Astronomy Village had a greater 
influence on Search for Lite content understanding than on problem solving. Therefore, the 
influence of the activity summary template on problem solving performance was limited. 

It is interesting to note that prior knowledge of the discipline, as measured by the pretest 
scores, was not a significant predictor of activity summary quality. The most significant predictors 
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were knowledge of cognition and problem representation. These results highlight the promise of 
this formative assessment approach for enabling low-achieving students to perform well in inquiry- 
based learning environments. Prior research has demonstrated that low-achieving students with high 
metacognitive self-regulation can perform equivalent to or better than high-achieving students who 
are low on metacognitive self-regulation (Howard, McGee, Shia, & Hong, 2001). This research 
helps to clarify the mechanisms by which metacognitive self-regulation can enable low-achieving 
students to perform well in inquiry-based learning environments. 

The results of the study point to the importance of helping students to process the 
information that they learn from the activities they engage in. The activity summary template seems 
to hold great promise as a classroom assessment tool to monitor how well students are developing a 
well-organized knowledge base that will assist them in problem solving. 
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APPENDIX A 



Name Brad W iters Date Oct 23. 1999 



Name of Activity Looking for Life in all the Right Places 



Activity Summary - Good Example 



I . Main Idea: Please write 3 sentences about the main ideas for this activity. Try to concentrate on 
the ideas that are most important to the question of finding life in the solar system. 



One of the main ideas in this packet is can life exist in an environment where it is cold, hot, salty or 
acid like conditions. Also if organisms can live on harsh conditions on earth could there have been 
or is there life on other planets. 



2. Importance : Please write 2 sentences about how this activity helps in your investigation. 



The connection between this, looking for life packet activity, and the Search for life core research 
question is how different life forms or organisms or anything living can live in or not live in harsh 
or perfect conditions. Also another connection is to use the information that scientists have to find 
out whether or not there is life on other planets. 



' 10 
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Name Kirby Kenwood Date Oct 22. 1999 

Name of Activity Looking for Life in all the Right Places 



Activity Summary - Poor Example 



I . Main Idea: Please write 3 sentences about the main ideas for this activity. Try to concentrate on 
the ideas that are most important to the question of finding life in the solar system. 



One thing we learned about was hyperthermophiles. We also learned where hyperthermophiles live 
and that they live in very hot temperatures. One last thing is we learned about phycrophiles and that 
they love cold. 



2. Importance : Please write 2 sentences about how this activity helps in your investigation. 



The connection is that we know that things can live in temperatures of 2 1 I degrees Fahrenheit and 
so cold as 0 degrees Celsius and they can be even microscopic. So they might even live in places we 
never thought of. 
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